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ABSTRACT 


Emotions are biological states associated with the nervous system, especially 
the brain brought on by neurophysiological changes. They variously cognate 
with thoughts, feelings, behavioural responses, and a degree of pleasure or 
displeasure and it exists everywhere in daily life. It is a significant research 
topic in the development of artificial intelligence to evaluate human behaviour 
that are primarily based on emotions. In this paper, Deep Learning Classifiers 
will be applied to SJTU Emotion EEG Dataset (SEED] to classify human 
emotions from EEG using Python. Then the accuracy of respective classifiers 
that is, the performance of emotion classification using Convolutional Neural 
Network (CNN] and Recurrent Neural Networks are compared. The 
experimental results show that RNN is better than CNN in solving sequence 
prediction problems. 
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INTRODUCTION 

Emotions play a significant role in how we think and behave 
in daily life. They can compel us to take action and influence 
the decisions we make about our lives, both large and small. 
As technology and the understanding of emotions are 
progressing, there are growing prospects for automatic 
emotion recognition systems. There exists a successful 
breakthrough in facial expressions or gestures as simulative 
emotional recognition. As an advanced machine learning 
technique for emotion classification, neural network is 
considered as a machine used to simulate how the brain 
performs a specific task. It stimulates the brain as a concept 
of complex nonlinear and parallel computers, which can 
evaluate complex functions according to various factors. A 
new direction this paper is heading towards is EEG-based 
technologies for automatic emotion recognition, as it 
becomes less intrusive and more affordable, leading to 
ubiquitous adoption in healthcare applications. In this paper 
we focus on classifying user emotions from raw 
Electroencephalogram (EEG] signals, using various neural 
network models and advanced techniques. We particularly 
explore end-to-end deep learning approaches, CNN and RNN, 
for emotion classification from SEED dataset directly. 
Convolution neural networks are forward neural networks, 
generally including a primary pattern extraction layer and 
feature mapping layer, and can learn local patterns in data 
by a method called convolution. We found that CNN [1] is 
better, and deep neural models are more preferable in 
emotion classification and estimation for machine computer 
interface system that models brain signals for emotions. 
However, an RNN remembers the entire information 
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through time. It is useful in time series prediction because it 
can remember previous inputs very well which is called 
Long Short-Term Memory. Also, recent developments in 
machine learning show that neural networks provide 
considerable accuracy in a variety of different tasks, such as 
text analysis, image recognition, speech analysis and so on. 

Related Works 

Automated classification of human emotion using EEG 
signals has been researched upon meticulously by various 
scholars. In related studies, there were many strategies 
proposed to perform emotion classification from EEG 
signals. It ranges from finding physiological patterns of 
emotions, identifying remarkable features, and any other 
combination of those strategies. 

In [1] the research provides a concise evaluation for CNN 
classifier which was reasonably accurate in identifying the 
two criteria of pleasure (Valence] and arousal degree 
(Arousal] with an average accuracy of 84.3% and 81.2%, 
respectively but for an individual subject, the average 
classification accuracy was only 63%. 

In [2] the paper showed, for both Valence and Arousal, DNN 
performs maximum classification accuracies of 75.78 and 
73.281 respectively while their CNN model had classification 
accuracies of 81.41% and 73.35% respectively. 

In [4] the research proved that the convolutional networks 
in comparison with the classic algorithms of machine 
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learning demonstrated a better performance in the emotion 
detection in physiological signals, despite being conceived 
for the object recognition in images. 

For classification of emotion, the most popular method, K- 
Nearest Neighbour algorithm [5] had achieved 62.3% overall 
accuracy by applying features namely, wavelet energy and 
entropy. The results showed 78.7±2.6% sensitivity, 
82.8±6.3% specificity and 62.3±1.1% accuracy on the 'DEAP' 
database. 

In [6] the paper showed that Hierarchical Structure-Adaptive 
RNN (HSA-RNN] for video summarization tasks, can 
adaptively exploit the video structure and generate the 
summary simultaneously. 

In [7] the results showed that the proposed framework 
improves the accuracy and efficiency performance of EEG 
signal classification compared with traditional methods, 
including support vector machine (SVM], artificial neural 
network (ANN], and standard CNN by 74.2%. 

In [8] the research proved that the TranVGG-19 obtains a 
good result with mean prediction accuracy is 99.175%. 

In comparison to SqueezeNet, Residual Squeeze VGG16 of [9] 
can be more easily adapted and fully integrated with residual 
learning for compressing other contemporary deep learning 
CNN models since their model was 23.86% faster and 88.4% 
smaller in size than the original VGG16. 

In paper [10], it was proved that the feature of mean gives 
the highest contribution to the classification using deep 
learning classifier, Naive Bayes with the highest 
classification result of reached 87.5% accuracy of emotion 
recognition. 

Proposed Work 

A. Database Agglomeration and Description 

The proposed method is conducted by SJTU Emotion EEG 
Dataset [SEED] which is a collection of EEG dataset provided 
by the Brain-like Computing and Machine Intelligence 
(BCMI) laboratory. This dataset contains the subjects' EEG 
signals when they were watching film clips. The film clips are 
prudently selected to induce different types of emotion, 
which are neutral, sad, fear and happy. 15 subjects (7 males 
and 8 females; Mean: 23.27, STD: 2.37] watched 15 video 
clips in the experiments. Each subject is experimented three 
times in about one week. The down-sampled, pre-processed 
and segmented versions of the EEG data in Matlab (.mat file] 
were obtained [2]. The detailed order of a total of 62 
channels is included in the dataset. The EEG cap according to 
the international 10-20 system for 62 channels is shown 
figure [1]. 



Fig.l EEG cap according to the international 10-20 
system for 62 channels 


The differences and ratios between the differential entropy 
(DE] features of 27 pairs of hemispheric asymmetry 
electrodes give the differential asymmetry (DASM] and 
rational asymmetry (RASM] features. By using the 
conventional moving average and linear dynamic systems 
(LDS] approaches, all the features are further levelled. 

B. Methodology 

The project exhibits emotion recognition using EEG signals 
to detect emotions, namely, happiness, sadness, neutral and 
fear using SEED dataset. Two different neural models are 
used, a simple Convolutional Neural Network and Recurrent 
Neural Network (RNN] as the classifiers. Both models are 
augmented using contemporary deep learning techniques 
like Dropout technique and Rectilinear Units in order to 
introduce non-linearity in our model. Both models are 
implemented using Keras [1] libraries in Python. 



Fig.2 Proposed Method 


RNN is a type of neural network which transforms a 
sequence of inputs into a sequence of outputs. An RNN can 
learn and detect an event, such as the presence of a 
particular expression, irrespective of the time, at which it 
occurs in a sequence. Hence, it naturally deals with a variable 
number of frames. 
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Fig.3 RNN Architecture 

At each time step t ; a hidden state ht is computed based on 
the hidden state at time t -1 and the input xt at time t 

ht = a (Win xt+Wrecht—V) 

where Win is the input weight matrix, Wrec is the recurrent 
matrix and a is the hidden activation function. Each timestep 
also computes outputs, based on the current hidden state: 

yt = f(Woutht) 

where Wout is the output weight matrix and f is the output 
activation function. 

We use a simple RNN with Rectified Linear hidden Units 
(ReLUs). We train the RNN to classify the emotions in a video 
agglomerated in the SEED dataset. Then using the confusion 
matrix, we measure the performance of each classifier in 
terms of accuracy, specificity, sensitivity, and precision. 


C. Confusion Matrix Parameters 

Confusion Matrix is a performance measurement table for 
machine learning classification where output can be two or 
more classes. It is composed of 4 different combinations of 
predicted and actual values. 

Experimental results and analysis 

RNN model is applied to construct EEG based emotion detection of four classes, namely Neutral, Sad, Fear and Happy. The 
dataset used here is SEED. The 62-channel EEG signals are recorded from 15 subjects while they are watching emotional film 
clips with a total of 24 trials. This data set is available with extracted features like Moving Average and PSD of Differential 
Entropy and LDS, differential and rational asymmetries, and caudal differential entropy. Further, these feature data belong to 
alpha and beta frequency bands are normalized with mean values and these have been used here. 

Tablel. Confusion matrix parameters for CNN and RNN 
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Fig.4 Confusion Matrix 

True Positive (TP): Prediction is positive and it's true. 

True Negative (TN): Predicted is negative and it's true. 

False Positive (FP) - Type 1 Error: Predicted is positive 
and it's false. 

False Negative (FN) - Type 2 Error: Predicted is negative 
and it's false. 


Accuracy: 

ACC = 


TP + TN 


TP + TN + FN + FP 


x 100% 


Specificity: 

SP= ™ 


TN + FP 


Sensitivity: 

TP 

SN = 


TP + FN 

Precision: 


PREC = 


X 100% 


X 100% 


TP 


TP 4- FP 


X 100% 



Confusion parameters CNN RNN 


Neutral 

Sad 

Fear 

Happi 

Hr 

Neutral 

Sad 

Fear 

Happy 

Accuracy 

95.53 

88.13 

88.01 

87.89 

96.50 

89.88 

90.85 

89.61 

Specificity 

98.56 

98.29 

57.44 

55.52 

93.44 

98.11 

74.18 

77.17 

Sensitivity 

94.54 

83.57 

96.68 

97.20 

98.29 

84.01 

98.14 

98.34 

Precision 

85.50 

72.85 

83.07 

85.08 

94.38 

81.37 

90.43 

88.74 


With these features of preprocessed EEG data, the existing 
CNN and proposed RNN models are trained. Then 
classification is performed using trained models. The 
experimental results show that the pools of features that 
have been chosen can achieve relatively classification 
performance across all the experiments of different subjects. 


From table (2) we see that RNN achieves the highest 
accuracy of 96.50%, 89.88%, 90.85% and 89.61% for 
Neutral, Sad, Fear and Happy while CNN model-based 
classification output gives 95.53%, 88.13%, 88.01% and 
87.89% for the respective emotions. 
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A. CNN 

For CNN, figure 5 shows the exponential behaviour of the 
accuracy during the training and testing for the 20 epochs. 
Similarly, in figure 6, the values of loss are displayed during 
the learning and testing that is decreasing for each epoch. 
The confusion matrix is showing the results of prediction for 
the four classes, respectively (figure 7). 
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Fig.7 Confusion matrix of CNN for the prediction of 
four emotions 


B. RNN 

For RNN, figure 8 shows the exponential behaviour of the 
accuracy during the training and testing for the 20 epochs. 
Similarly, in figure 9, the values of loss are displayed during 
the learning and testing that is decreasing for each epoch. 
The confusion matrix is showing the results of prediction for 
the four classes, respectively (figure 10). 



Fig.9 Loss result for RNN 
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Fig.10 Confusion matrix of RNN for the prediction of 
four emotions 
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Hence, this paper shows that RNN figure (10) has made the 
highest prediction for the four classes compared to CNN 
figure (7). 

Conclusion: 

Based on the existing research in the field of emotion 
recognition, this study explores the effectiveness of the 
recurrent neural network. The experimental results show 
that the RNN model achieves higher precision to classify 
emotions with CNN based approach. The reliability of 
classification performance suggests that specific emotional 
states can be identified with brain activities. The learning by 
RNN suggests that neural signatures associated with Neutral, 
Sad, Fear and Happy emotions do exist and they share 
commonality across individuals. Thus, RNN provides an 
appealing framework for propagating information over a 
sequence using a continuous-valued hidden layer 
representation. 
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